fix(audit): surface pre-agent step errors when agent never executed#17623
fix(audit): surface pre-agent step errors when agent never executed#17623
Conversation
…ent never ran
When a workflow run fails in a pre-agent step (e.g., lockdown validation,
missing secrets), the audit tool now extracts the actual error message from
the step log files in workflow-logs/{job}/{step_num}_{step_name}.txt instead
of reporting "No specific errors identified".
Also improves the error message for invalid run IDs to provide a human-readable
message instead of the cryptic "exit status 1".
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes a significant gap in the audit command where failed workflow runs that never executed the agent step reported "No specific errors identified" despite the actual error being present in downloaded step log files. The fix extracts and surfaces the actual failure messages from GitHub Actions step logs when the agent never ran.
Changes:
- Adds extraction of pre-agent step errors from workflow-logs when agent-stdio.log is absent and workflow failed
- Implements timestamp stripping for GitHub Actions log format to provide clean error messages
- Improves error messaging for non-existent run IDs with actionable guidance
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/cli/audit_report.go | Implements extractPreAgentStepErrors to scan step logs, parseStepFilename to parse step numbering, stripGHALogTimestamps to clean log output, and integrates error extraction into buildAuditData |
| pkg/cli/audit_report_test.go | Comprehensive test coverage for all new functions including edge cases, timestamp formats, truncation, and integration with buildAuditData |
| pkg/cli/audit.go | Adds human-readable error message when workflow run ID is not found instead of generic "exit status 1" |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| var lastStep *stepLog | ||
|
|
||
| jobDirs, err := os.ReadDir(workflowLogsDir) | ||
| if err != nil { | ||
| return nil | ||
| } | ||
|
|
||
| for _, jobEntry := range jobDirs { | ||
| if !jobEntry.IsDir() { | ||
| continue | ||
| } | ||
| jobDir := filepath.Join(workflowLogsDir, jobEntry.Name()) | ||
| stepFiles, err := os.ReadDir(jobDir) | ||
| if err != nil { | ||
| continue | ||
| } | ||
| for _, stepFile := range stepFiles { | ||
| if stepFile.IsDir() || !strings.HasSuffix(stepFile.Name(), ".txt") { | ||
| continue | ||
| } | ||
| num, stepName := parseStepFilename(stepFile.Name()) | ||
| if num > 0 && (lastStep == nil || num > lastStep.num) { | ||
| lastStep = &stepLog{ | ||
| path: filepath.Join(jobDir, stepFile.Name()), | ||
| num: num, | ||
| stepKey: jobEntry.Name() + "/" + stepName, | ||
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The current logic finds the step with the highest step number across ALL jobs in workflow-logs. In a multi-job workflow where a job fails early (e.g., "setup" fails at step 5) but another job successfully runs later steps (e.g., "build" completes steps 1-15), this would extract content from the "build" job instead of the failing "setup" job.
While this works correctly for single-job workflows (which appears to be the primary use case based on the issue description mentioning "the agent job"), it may produce misleading results in multi-job scenarios.
Consider documenting this limitation in the function comment, or as a future enhancement, filtering to only examine jobs that failed (though this would require passing job information to this function).
When a workflow fails before the agent runs (lockdown validation, missing secrets, binary install failures),
auditreports"No specific errors identified"despite the actual error being present in downloaded step log files. The errors array was always empty since agent-stdio.log-based extraction was removed, leaving no fallback.Changes
pkg/cli/audit_report.goextractPreAgentStepErrors(): whenagent-stdio.logis absent and workflow conclusion isfailure, scansworkflow-logs/{job}/{step_num}_{step_name}.txtand returns the last executed step's content (highest step number = likely failing step)parseStepFilename(): parses12_Validate lockdown mode requirements.txt→(12, "Validate lockdown mode requirements")stripGHALogTimestamps(): strips2024-01-01T10:00:00[.sss...]Zprefixes from log lines; handles variable-length fractional secondsbuildAuditData(): callsextractPreAgentStepErrorsfor failed runs, feeding the result intogenerateFailureAnalysissoerror_summaryreflects the actual failurepkg/cli/audit.gofetchWorkflowRunMetadata(): returns a human-readable error for non-existent run IDs instead of propagating the rawexit status 1Before / After
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GO111MODULE 64/bin/go infocmp -1 xterm-color go 3313510/b430/workflow.test -json GO111MODULE 64/bin/go 3313510/b430/workflow.test(http block)https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha jpq_JbW5I GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go l GOMOD GOMODCACHE ortcfg env g/timeutil/format.go g/timeutil/format_test.go /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v3/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha 7848/001/stability-test.md GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 1520-27702/test-1652831907/.github/workflows GO111MODULE .cfg GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha 01 GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env runs/20260222-001520-27702/test-3457897160/.github/workflows GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha -json rsion=abf14b3-dirty /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE .cfg GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha uts.version Test User /usr/bin/git -json GO111MODULE 64/bin/go git rev-�� --show-toplevel node /opt/hostedtoolcache/node/24.13.0/x64/bin/node --check **/*.cjs 64/bin/go node(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v5/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha ithub/workflows/bot-detection.md GOPROXY /usr/bin/git GOSUMDB GOWORK 64/bin/go git rev-�� s/test.md go /usr/bin/git -json GO111MODULE 64/bin/go git(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha user.name Test User /usr/bin/git che/go-build/d7/git GOPROXY 64/bin/go git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.0/xGO111MODULE /usr/bin/gh /tmp/go-build248git -trimpath 64/bin/go gh(http block)https://api.github.com/repos/actions/github-script/git/ref/tags/v8/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go env ck 'scripts/**/*GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha go1.25.0 -c=4 -nolocalimports -importcfg /tmp/go-build2203313510/b389/importcfg -pack /tmp/go-build2203313510/b389/_testmain.go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD erignore go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/setup-go/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha i/install.sh GO111MODULE /opt/hostedtoolcache/go/1.25.0/x-nilfunc GOINSECURE GOMOD GOMODCACHE go env 1520-27702/test-1895399 GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go l GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha /ref/tags/v8 GO111MODULE Name,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts/usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE ode_modules/.bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go estl�� ty-test.md GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE _modules/.bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE debian-sa1 estl�� 1 GOPROXY 64/bin/go GOSUMDB GOWORK modules/@npmcli/add go(http block)/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE es/.bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go N files are not git GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts/usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env ty-test.md GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE odules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts/usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE _modules/.bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts/usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE es/.bin/node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts/usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE node GOINSECURE GOMOD GOMODCACHE go ode_�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/workflows/usr/bin/gh gh workflow list --json name,state,path GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node(http block)/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env 3603486071/.github/workflows GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999/usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/nonexistent/repo/actions/runs/12345/usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE go env th .prettierignore GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/owner/repo/actions/workflows/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env tformat GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go /home/REDACTED/worgo go /usr/bin/git sh -c "prettier" --wriGOINSECURE git 64/bin/go --show-toplevel x_amd64/compile /usr/bin/git go(http block)https://api.github.com/repos/owner/repo/contents/file.md/tmp/go-build2203313510/b380/cli.test /tmp/go-build2203313510/b380/cli.test -test.testlogfile=/tmp/go-build2203313510/b380/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD ode-gyp-bin/sh go k/gh�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD erignore go(http block)/tmp/go-build290692283/b001/cli.test /tmp/go-build290692283/b001/cli.test -test.testlogfile=/tmp/go-build290692283/b001/testlog.txt -test.paniconexit0 -test.timeout=5m0s -test.run=^Test -test.short=true security 64/bin/go OUTPUT -d 168.63.129.16 node /hom�� --write **/*.cjs 64/bin/go **/*.json --ignore-path run-script/lib/n"prettier" --check '**/*.cjs' '**/*.ts' '**/*.json' --ignore-path ../../../.prettierignore sh(http block)https://api.github.com/repos/test-owner/test-repo/actions/secrets/usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go m/_n�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name --noprofile git 64/bin/go --get remote.origin.urenv /usr/bin/git sh -c "prettier" --wriGOINSECURE git 64/bin/go /ref/tags/v8 go /usr/bin/git make(http block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.